Environmental and computing cost reduction with improved reliability in workload assignment to distributed computing nodes

ABSTRACT

A system and method of allocating a job submission for a computational task to a set of distributed server farms each having at least one processing entity comprising; receiving a workload request from at least one processing entity for submission to at least one of the set of distributed server farms; using at least one or more conditions associated with the computational task for accepting or rejecting at least one of the server farms to which the job submission is to be allocated; determining a server farm that can optimize the one or more conditions; and dispatching the job submission to the server farm which optimizes the at least one of the one or more conditions associated with the computational task and used for selecting the at least one of the server farms.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to a global job distribution system andmore particularly to assigning computational tasks according to aselectable set of criteria.

2. Description of the Related Art

Currently large national and multinational corporations rely on high-endcomputing power to enable and/or efficiently operate their businesses.To satisfy the continually increasing demand for computing power inthese organizations, sets of servers and memory units are groupedtogether into clusters which are commonly known as server farms. Theseserver farms, with their concentration of high performance CPUs, requiresignificant amounts of electrical energy to operate. Energy demand forthese server farms is further increased by operating the airconditioning or other mechanical chilling/cooling systems to keep thecomputing equipment cool. Server energy demand has driven at least onelarge scale data processing company to locate their newest server farmsclose to an economical power source such as a hydroelectric dam.

While multi-national companies operate clusters or farms of servers atmultiple locations in the U.S. and throughout the world to handle dataprocessing workload, a job assignment to a particular server in aparticular cluster is still largely performed on a local basis where thejob submitter selects a particular cluster by logging into the cluster,or is part of a group of users assigned to a cluster by geographicalregion. This workload may come from within a company, from customersaccessing company databases via the Intranet or Internet or throughdata-processing lease agreements where a company without servers leasesserver capability from an enterprise company. Recently, some attentionwithin the data processing community has been devoted to allocating dataprocessing resource on a global level in order to optimally utilizeavailable resource. In those instances where the workload on the serversis considered when assigning data processing or other computer jobs to aparticular server node and server, no consideration is given in the jobdispatching process to the sources of energy which will be used to powerthe server farm (coal, oil, nuclear, hydro, solar), the local cost ofenergy or the condition of the local energy grid. As a result, jobdispatch is not conducted in a manner which can minimize operationalcosts and/or use the most environmentally friendly energy sources whileperforming the necessary data processing workload. Furthermore, companydata processing operations may come into direct conflict with the needsof the local community during those periods when the electrical grid isunder stress.

SUMMARY OF THE INVENTION

To overcome the shortcomings noted above, there is disclosed a globalworkload assignment algorithm which allows for at least the utilizationof the most environmentally friendly power source while processing data,that can balance a customer's requirement for speed in processing datawith cost and environmental objectives, that allows for increasing powerneeds of a data processing center to be adjusted for unusual localconditions which may impact both data processing reliability andcommunity wellness, and/or provide a framework for competitiveness inthe area of data processing.

In an embodiment there is disclosed a method of allocating a jobsubmission for a computational task to a set of distributed server farmseach having at least one processing entity comprising;

receiving a workload request from at least one processing entity forsubmission to at least one of the set of distributed server farm;

using at least one or more conditions associated with the computationaltask for accepting or rejecting at least one of the server farms towhich the job submission is to be allocated;

determining a server farm that can optimize said one or more conditions;and

dispatching the job submission to the server farm which optimizes the atleast one of the one or more conditions associated with thecomputational task and used for selecting the at least one of the serverfarms.

In another embodiment there is disclosed a system of allocating a jobsubmission for a computational task to a set of distributed server farmseach having at least one processing entity comprising:

receiving means for receiving a workload request from at least oneprocessing entity for submission to at least one of the set ofdistributed server farms;

selecting means using at least one or more criteria or conditionsassociated with the computational task for accepting or rejecting atleast one of the server farms to which the job submission is to beallocated;

determining means for providing a server farm that can optimize said oneor more conditions; and

sending means for dispatching the job submission to the server farmwhich optimizes the at least one of the one or more conditionsassociated with the computational task and used for selecting the atleast one of the server farms.

In still another embodiment there is disclosed a computer programproduct for use with a computer, the computer program product includinga computer readable medium having recorded thereon a computer program orprogram code for causing the computer to perform a method for storingand retrieving data, the method comprising:

receiving a workload request from at least one processing entity forsubmission to at least one of the set of distributed server farms;

using at least one or more conditions associated with a computationaltask for accepting or rejecting at least one of the server farms towhich the job submission is to be allocated;

determining a server farm that cam optimize said one or more conditions;and

dispatching the job submission to the server farm which optimizes the atleast one of the one or more conditions associated with thecomputational task and used for selecting the at least one of the serverfarms.

The foregoing has outlined, rather broadly, the preferred feature of thepresent invention so that those skilled in the art may better understandthe detailed description of the invention that follows. Additionalfeatures of the invention will be described hereinafter that form thesubject of the claim of the invention. Those skilled in the art shouldappreciate that they can readily use the conception and specificembodiment as a base for designing or modifying the structures forcarrying out the same purposes of the present invention and that suchother features do not depart from the spirit and scope of the inventionis its broadest form.

BRIEF DESCRIPTION OF THE DRAWINGS

Other aspects, features, and advantages of the present invention willbecome more fully apparent from the following detailed description, theappended claims, and the accompanying drawings in which similar elementsare given similar reference numerals.

FIG. 1 is a view of a high level architecture in which a Global WorkloadAssignment Engine (GWAE) communicates with a multiplicity of serverfarms which may be in various geographic locations and maintains a jobqueue for the assignment of workload to the server farms;

FIG. 2 is block diagram of a Global Workload Assignment Engine (GWAE);

FIG. 3 is a flow chart of the function for weather mitigation;

FIG. 4 is a flow chart of the function of monitoring workload and energyconsumption;

FIG. 5 is a chart of information for real time monitor of power,workload, job priority in different server farms as well as cost andgreen rating determination;

FIG. 6 is a flow chart of a mitigator function which analyzes processingrequests from multiple submitters and sends a request-for-bid to each ofthe entities controlling processing resource; and

FIG. 7 is a flow chart of the function for handling carbon credittrading.

DETAILED DESCRIPTION OF THE INVENTION

There is disclosed a global job and distributed submission system whichassigns, and continually reassigns as needed, computational tasksaccording to a set of criteria of arbitrary number in which one of moreof the following are included to manage a complex set of factors whichare used for the computation of a computer job assignment. These factorscan include environmental cost of power used at all server nodes,knowledge of current carbon credits and the difference in energyconsumption, access to weather data bases, access to power supplierdatabases, and locally specified data compiled at each node which isconsidered important to the institution (business or other agency).

The computation assignment “engine” and algorithm embodied in programinstructions that are executed herein continually examines a variety offactors, some of which are fixed prior to job submission, and others ofwhich are in continual flux. If selected factors go outside apredetermined range for a specified factor, then any or all jobs may bereassigned at that time.

The assignment engine may operate to assign and dispatch workload, andmonitor completion and reliability in the following areas:

Environmentally Conscious (Green) Workload Assignment:

The engine works to assign workload to one or more server farms thathave the lowest environmental impact as measured by the power sourceused to power the farm, the server farm power performance product, theexpected processing time of the workload and energy requirements fortransporting the workload to the processing site.

Weather Risk Mitigation:

The engine may further link with other databases to determine whencertain criteria, or conditions have occurred or are about to occur atthe server farm location. For example, the farm's processingfacilities/resources become at risk due to incipient weather eventswhich may interrupt processing capabilities and/or communication withthe processing site. Database links may include an official or privateweather forecasting database that routinely issue watches and warningswhen severe weather is expected. The engine may down-grade the effectedprocessing resource such that a new workload will not be assigned to theat risk resource and/or the engine may move a presently assignedworkload which may be exposed to the event and which has a long run timeexpectation to an alternate processing location.

Global Security Risk Mitigation:

The engine may also be linked to a security data base(s) such as thatsupplied by US Homeland Security or individual business firms. This canbe used to direct, in a weighted fashion, the processing power to orfrom areas of relative safety or temporary instability, respectively. Asworld security conditions dynamically change, this mitigation may beused to enhance or override the other factors feeding data to the globalworkload assignment engine.

Grid Risk Mitigation:

In certain instances, such as on warm summer days when total demand on apower supplier may approach the capacity of the system and transmissioncapability is at its limit, the power supplier may be forced to bring online a less environmentally friendly generating unit and/or power supplyinterruptions may occur with resulting data loss. When such conditionsexist, the engine may further link with a supplier database to determinewhen generating and transmission resources are approaching their limitsand work to divert or re-assign the processing workload to reduce powerrequirements and improve processing reliability.

Cost Competitiveness:

In a shared network where many processing providers (not just onecompany) have server farms that may take in one or more workloads fromthe engine, the engine may further receive data from each processingprovider about processing costs in addition to capability, availabilityand efficiency. These costs may be constant or updated according to thetype of workload and organization requesting the processing capacity,etc. The engine may operate to solicit bids for a workload and factor inthe bid price for the workload, along with other factors such as theenvironmental impact, the expected job completion time, etc. in a mannerwhich will manage its operation and balance the cost and/orenvironmental impact.

Carbon Trading Support:

The engine may further integrate the assignment of a workload to a“green” server resource with a carbon trading system which calculatescarbon credits as a function of CO and/or CO₂ level differentialsbetween the energy source used and a base CO₂ level used to calculate acarbon credit/debit.

Referring to FIG. 1, there is shown a high level architecture in which aGlobal Workload Assignment Engine (GWAE) 10 communicates with amultiplicity of server farms 12 which may be in various geographiclocations and maintains a job queue 18 including one or more submittedwork requests for the assignment of at least one workload to the serverfarms. Each server farm may have multiple computing devices, e.g.,servers, each having varying processing capabilities, utilizations,costs and environmental footprints. Likewise, power providers 14 whichprovide the primary energy source for each farm may have differentgeneration capability, utilization, cost, environmental footprint andinterruption risk. The GWAE, through an interface with a number ofdifferent databases 16, considers the environmental footprint,processing and transmission costs along with typical factors in workloadassignment such as processing availability and capability to assign aworkload to each server farm in a manner which minimizes theenvironmental footprint, cost, or both. Further the GWAE monitorschanging conditions to determine when work in progress should be movedfrom one server farm to another to further reduce the environmentalfootprint and/or cost, or to avoid a potential loss of work in progress.In an embodiment it could be set up as a privately run, publicaccessible resource, or it could be established by a private institutionand leased.

Referring to FIG. 2, there is shown a block diagram of a Global WorkloadAssignment Engine (GWAE) 10 in accordance with the principles of theinvention. The engine dispatches workloads that may be generated atmultiple sites/geographies to a number of distributed server farm nodes12 for execution. A site submitting workload to the GWAE may or may nothave native processing resource to execute the workload. Should thesubmitting site have native capability, the GWAE considers nativeprocessing as one alternative in making the workload assignment.

Outgoing workload from each site 22 to the GWAE 10 contains a parameterfile which contains parameters including, but not limited to: submitteridentification attributes, outgoing 24 and incoming 26 data/filelocations, processor requirements, memory requirements, expectedexecution time, software requirements, etc., used to place a workload oncompatible execution platforms. As part of the method taught, theparameter file includes one or more “green” parameters 28 used to drivethe assignment of workloads within a spectrum of server farm optionsthat may contain both a mix of friendly and unfriendly environmentalfootprints. Green parameters weigh the selection of processing resourcestowards high performance capability, low cost capability orlow-environmental impact capability. At its extreme, green parametersmay be used to restrict processing to nodes with the lowestenvironmental impact. Regardless of the green parameter setting, theGWAE 10 operates to provide the lowest environmental impact whilesatisfying operational, performance and cost constraints.

Workload outgoing from each submitting site may be optionally audited toinsure that “green” parameters match the range of settings allowed as adepartment, project or corporate policy. When implemented, this auditmay be conducted by respective processing of data in blocks 30, 32, 34on conductors 36, 38, and from 40, 42 and 44 shown in FIG. 2 in theoutgoing data-stream from the submitting location, or as an auditprocessing module within the GWAE engine itself. In the former case, thesite locally controls and updates audit settings for the data stream,and in the latter case, the department, project or company loadsparameters to the GWAE and the GWAE differentiates between workloadsbased on identifier parameters.

The GWAE may operate to initially only receive the parameterinformation, perform the execution assignment and facilitate data/filecommunication directly between the submitting and executing sites.Alternatively, all required files may be sent to the GWAE in the initialtransfer with the GWAE managing file transfer between the submitting andexecuting sites.

As additional inputs to the assignment process, the GWAE also receivesinput 36, through database query or from other means on each server farmavailable for workload assignment including, for example, server farmcapacity availability, performance capability and energy efficiency.Energy efficiency data may include not only the energy demand/efficiencyof the computer servers (primary energy consumption) but thedemand/efficiency of the physical plant which supports the computerservers (air conditioners, water chillers, etc.). Database access may beprovided in a web-based environment, by a dedicated link or by othermeans.

The GWAE also receives input 38, though database query or other means,information on the condition of each power provider which serves one ormore server faun nodes in the GWAE system. Information continuouslygathered may include, for example, the present load vs. capacity on theproviders generation and transmission facilities as well as the presentmix of power generation (i.e., fossil fuel, nuclear, solar, hydro, etc.)with percentages. Database access may be provided in a web-basedenvironment, by dedicated link, or by other means. The GWAE may furtherquery a power supplier—server farm cross reference which may be constantor continuously updated to determine which suppliers are contributingpower to each server farm if the relation is not implicit for any serverfarm.

In order to place workloads in a manner which minimizes energyconsumption and uses the most environmentally friendly power sources, anumber of sub-processes operate within the GWAE.

A job queue within the GWAE operates to handle incoming workloads, forexample, on a first-in-first-out (FIFO) basis, holding information onjob parameters, green and other information necessary for assignment.

The GWAE monitors parameters 40 from each server farm, calculates andupdates the energy consumption and environmental footprint perstandardized unit of processing capacity based on the performance ofavailable servers, the power those servers demand and the secondarypower required to run the facility. Server farms are ranked 44 accordingto energy consumption in conjunction with green parameterperformance/green weight in the parameter file. Further, the GWAEapplies the calculations to the expected processing requirements for therequest to calculate the total energy requirements. Expected processingrequirements may be entered as parameter inputs or calculated based onpast history for file sizes, software called, customer history, etc.Availability of servers within each server farm is used to furtheraugment the ranking such that workload is not assigned to nodes thatlack sufficient capacity.

While energy consumed may be principally driven by the executing serverfarm for large processing jobs, smaller jobs may use transmission energywhich is significant in proportion to the processing energy whenmovement of data across geographies is considered. Because of this theGWAE estimates the energy required to transfer workload from thesubmitting site to the executing site for each assignment possibility.For small jobs, less total energy may be consumed when jobs are executedlocally on less efficient systems than when the data is transmitted to adistant location.

In addition to ranking the server farms, the GWAE also ranks powersuppliers 42. Suppliers operating from renewable resources are rankedabove suppliers operating from fossil fuels. Where non-renewable sourcesare employed, the sources are ranked based on generation efficiency andenvironmental footprint per unit of energy produced. Where suppliersoperate using a mix or renewable and non-renewable sources, supplierranking is influenced by the percentage mix of the sources. Generationmix may further be used to calculate cost per unit of energy as well asenvironmental footprint for factoring into workload assignment. Rankingmay be further augmented using preferences within the spectrum ofrenewable sources. For instance, solar or wind energy may be preferredover hydro energy due to concerns over water reserves, etc.

The GWAE further correlates ranked server farm data 44, FIG. 2, withranked power supplier data from power suppliers 46, FIG. 2, to determinethe server farm which best meets the submission parameter criteria.Server farms with the lowest environmental impact may be identifiedbased on supplier and faint ranking/correlations 43 as well as datatransfer energy estimation. Likewise, solutions with the lowest cost orthe highest performance can be identified.

Workload assignment within the GWAE operates with ranking functions suchas the one described above in conjunction with the green parametersassociated with the workload. Workloads (jobs) marked for assignment ononly those computing resources which are 100 percent renewable will beplaced on nodes which have the required energy source profile. If noinitiators are available, the jobs will remain in the queue until aresource is available. A workload not marked for 100 percent renewablecomputation will be assigned to the most-environmentally friendlyresource available that meets the performance constraints at anassignment time. If a resource that is 100% renewable powered isavailable and no job requiring this profile exist in the queue, thisworkload is assigned to the fully renewable resource unless the resourcedoes not meet the performance or other workload requirements. Should anew workload requiring the renewable resource be submitted, a workloadwhich does not require fully renewable processing may be checkpointedand moved to the next available initiator as ranked. Checkpoint isdefined as Flagging a job so that it is an indication for the job to bemoved Similarly, a workload originally assigned to a lessenvironmentally friendly node by the GWAE may be moved to a morefriendly node should initiators become available. Movement decisions maybe based on an expected resource required to complete the workload asopposed to that required to checkpoint and transport the required filesto another node, or other factors as well as initiator availability. Asa default, the GWAE always works to assign a workload to the lowestenvironmental footprint solution considering energy consumption fortransport and processing as well as the energy mix used at theprocessing site (in addition to performance or other installationrequirements). For small jobs, the best solution as determined by theGWAE may be to run the job locally to avoid transport energyconsumption. As a default, the GWAE operates to minimize theenvironmental footprint for the workload unless the performance/costparameters make the execution on green resource impossible, in whichcase the GWAE operates to satisfy job parameters in the order of theirimportance as weighted in the parameter file.

Once an assignment is made, the GWAE either operates as an intermediarytransferring required files from the submitting to the executing site oroperates as a facilitator passing instruction to each of the submittingand executing sites for direct communication. Further, the GWAE maymonitor the server farm 40 and workload status 48, FIG. 2, andcommunicate with the submitter and executor to facilitate anycheckpoints or workload moves.

Weather Risk Mitigation

As disclosed above and shown in the flow chart of FIG. 3, the methodperformed by the GWAE utilizes a number of databases to incorporate realtime parametrics for server farms and power suppliers through web-basedlinks, dedicated links or by other means into the workload placementalgorithm. In another embodiment, the method may additionally compriseadaptive job assignment based on impending environmental events whichmay disrupt workload processing at any server farm or communicationbetween a farm and the GWAE and/or the submitting node. Examples ofenvironmental events that the GWAE may operate are watches, warnings,occurrences of hurricanes, tornados, flood or winter storms, althoughother events may be considered.

Referring to FIG. 3, to provide weather risk mitigation, flow chart 302shows how the GWAE links to a weather prediction database 304 managed bya public or private entity, examples of which are National WeatherService (NOAA), Environment Canada, etc. to retrieve regional forecastsfor geographic zones which contain server facilities and/or criticalpower production facilities. Each forecast may contain watches andwarnings for severe weather events which may affect the ability toprocess one or more workloads or deliver power to a server farm alongwith the event type, probability of the event, and time frame for theevent. The GWAE uses the additional information obtained from oraccessed via the weather risk link, along with other server farm powersupplier and job submission parameters to confirm initial job submissionassignment by the GWAE (step 306). The mitigation method may comprisecalculating expected completion time for workload on the selected node(step 308) and examining the forecast as provided by database 304 forthe estimated processing timeframe (step 310). Once the GWAE determinesthat workload processing time will not overlap an at risk time period(step 312) that workload is eligible for submission to the farm inquestion. Thereafter, it is determined if the weather risk has increasedbased on the completion time on the job (step 314). If yes (YES; step314), the job is checkpointed and potentially reassigned (step 316). If,however, weather risk to the workload has not substantially changed,(NO; step 314), the job is run on the present server to completion (step318) or until a degradation in the weather forecast is received (step314). Furthermore, weather information is routinely monitored for adefined set of adverse (or worsening) conditions, which would have thepotential to disrupt the data processing. These changes in weather, oncea workload is submitted to the farm and adverse changes in weathercondition, may trigger checkpointing and possible movement of a workloadto alternate farms. While the at-risk node may be under-utilized withregard to long running workload, the GWAE may preferentially route shortrun workloads to the at risk node in the interim period. The method mayfurther comprise auditing an existing work load on the server node todetermine if any workload is expected to still be running at the onsetof the risk period, determining whether to checkpoint and move anaffected workload from the affected node, and using the GWAE to reassignthe checkpointed workload. Reassignment may be performed using aseparate prioritized re-assignment queue within the GWAE or byprioritization within the submission queue of the GWAE. Additionally themethod may further comprise GWAE prediction of outage probability basedon the watch/warning presented, past history and, in some instances,more detailed weather data with a GWAE decision point for mitigationbased on prediction.

While not considered a weather event, and not always predictable, othernatural disasters such as earthquakes, volcanic activity and tsunamismay be mitigated in the same manner assessing risk time frame anddiverting workload to non-compromised processing resources.

Grid Risk Mitigation

As demands for energy increase faster than power suppliers can bring newpower generation and transmission facilities, suppliers have institutedprograms that notify the industry and the public during periods wherepeak energy requirements are estimated to be close to the maximumgeneration/transmission capability of the geographic area. The goal ofthese programs is to request conservation/reduction of utilization inorder to prevent either rolling blackouts or utilization of short-term,less-efficient (dirtier) power supplies which drive up thecost/environmental impact.

As a corollary, server farms themselves may face periods of time whenthe energy consumed by the servers and the physical plant (cooling, etc)approximate or exceed the intended maximum designed power capability forthe facility due to increases in capacity, transition to higher-powerservers, or summer temperature extremes. In each case, completion of theworkload currently running or queued for processing on the affectedserver farm becomes at risk.

Referring to FIG. 4, there is shown a flow chart of a function operation402 within SF1, there being a similar function operation within eachserver farm to monitor workload and energy consumption and communicatestatus back to the GWAE through one or more links. To pick the correctvariable from FIG. 5, for that SF, the Flow in FIG. 4 will actually havex2, dx2 and y2 for a SF2 flow. The flow in FIG. 4 will actually have xn,dxn and yn for SFn flow as defined in FIG. 5. Initially, after job 1 isadded to SF1 (step 404), the function executed at a device within SF1determines if y1 is greater than x1 where energy consumption (summationof processing energy, cooling energy, etc.) is monitored as y1 and whereX1 is the desired normal power consumption maximum for the facility. Ify1 is less than x1 (NO; step 406) the function proceeds to set SF1availability flag=1 to accept future jobs (step 408). Thereafter, thefunction processes jobs at optional processing capability (step 410),then accepts a next job if available (step 412), and returns to step406. If, however, at step 406, y1 is greater than x1 (YES; step 406),the process determines if y1 is greater than x1 plus dx1 where dx1represents the allowable short term power consumption above the normalmaximum x1. If it is not, (NO; step 414) the process sets SF1availability flag to “0” to not accept future jobs (step 416), thentransfers a low priority job to GWAE for reassignment (step 418) inorder to return to y1<x1, and returns to step 406. If, at step 414 thefunction determines that y1 is greater than x1 plus dx1 (YES; step 414),the process checkpoints all workloads to ensure loss of intermediateresults does not occur (step 420), then processes jobs at a lowerprocessing capability (step 422) and advances to step 416.

Restating the above described function of grid risk mitigation, aworkload assigned by the GWAE to a given SF1 is added to the processingworkload, and energy consumption is monitored as “y1”. If y1 is belowthe rated threshold “x1” for the facility, the SF communicates back tothe GWAE that it is available for an additional workload through theserver farm database (see FIG. 5, upper left block, lines 1 and 2). Oncethe amount of the workload forces y1 to be greater than threshold x1,the SF communicates back to the GWAE that it has reached/exceeded itsdesigned power consumption and that it is no longer available for anyadditional workload.

Energy consumption y1 is checked against a second threshold “x1+dx1” todetermine if the total energy consumption is below or above the worstcase (short term) maximum for the facility. If y1 is below x1+dx1 the SFin communication with the GWAE begins to checkpoint lower priorityworkload for re-assignment to other SFs by the GWAE. Prioritization ofthe workload is maintained in the job priority database (see FIG. 5,lower left block). If y1 is above x1+dx1 the SF begins to checkpoint allworkloads in order to ensure that intermediate results are not lost.Workloads may continue to be processed on the SF at lower processingcapability (speed) in order to reduce power consumption below the shortterm maximum as a lower priority workload is reassigned by the GWAE inorder to bring the server farm energy consumption to y1<x1.

Referring to the chart of FIG. 5, the GRID INFO DATABASE comprises 3databases—a server farm database, a Job priority data base and a Powerprovider database. Server Farm database keeps track of the maximum powerhandling capability, a tolerance indicating power level (red zone range)and Real time power consumption for EACH SF. Job Priority database keepstrack of the real time individual jobs being processed by each SF andthe priority of jobs under each SF. Power Provider database: the PPdatabase keeps track of EACH Power provider's (PP) Green rating,Availability status to each of the SF, real time power supplied to eachSF, Usage cost to each SF and power transmission cost to each SF. Thedatabase also keeps track of Each SF's total power contributed by thedifferent PPs, Net green rating of each SF, and Net usage & transmissioncost of each SF. To insulate the workload from power supply/transmissionstresses, the databases from which the GWAE gets parameters for workloadassignment are augmented to include messaging fromsuppliers/transmission operators on conditions and forecasting as thedigital equivalent of “electrical load day” designations commonlyprovided to inform power customers of peak loading conditions. The powerprovider database provides, along with green rating % and costinformation used in workload assignment above, information on powersupply/transmission availability (capability, use) for each of the oneor more power providers for each server farm. When powersupply/transmission margins reach a threshold, the GWAE may act toprevent additional workload on the SF in question and possibly move oneor more workloads in progress to other SFs to reduce the load on thesuppliers/grid and insure that the workloads are completed.

Referring to FIG. 5, there is shown an information chart of an exampleof the result of a power provider ranking algorithm calculation of NetGreen Rating (NETGR_SF1) for Server Farm One (SF1) where

-   -   PP1 with 50% Green Rating provides 300 Kwatts to SF1;    -   PP2 with 20% Green Rating provides 200 Kwatts to SF1;    -   PP3 is not available for SF1; and    -   PP4 with 10% Green Rating provides 500 Kwatts to SF1.

Thus, the total power supplied by three Power Providers, PP1, PP2 andPP4 to Server Farm one (SF1) is NETPWR_SF1=300K+200K+0+500K=1000 Kwatts.

The Green Rating (GR) of SF1=NETGR_SF1=(300 Kwatts/1000 Kwatts)×50%+(200Kwatts/1000 Kwatts)×20%+(500 Kwatts/1000 Kwatts)×10%=15%+4%+5%=24%

Cost Competitive Environment

The GWAE is not limited to distributing the workloads amongst serverfarms controlled by a single public or private entity, but may beexpanded to perform gateway services to server farms owned by multiplecompanies or governments. When heterogeneous ownership of processingcapacity exists, it is recognized that management of each server farmmay be done in a competitive manner to obtain workload share. To enablecompetition between multiple server farm owners in a GWAE environment,the GWAE is programmed to provide a mitigator function 602 (FIG. 6). Themitigator function (at step 608) analyzes processing requests frommultiple submitters (requestor(s) 604) and sends a request for bid toeach of the entities controlling processing resources (server(s) 610).The request may contain information on hardware and softwarerequirements, job size, performance, resource and/or green constraints.Each entity (server(s) 610) may bid on the workload in an effort tosecure it. The bid may include quotes for one or more of processingcost, green processing capability, information transmission cost,processing time, or carbon credit earnings. The GWAE may select (step612) a processing provider (server choice 616) for each requestor 614for the subject workload based on any one factor or a weighting offactors to satisfy the workload submitter's requirements and optimizethe processing constraints to the submitter's requirements. The GWAEmitigate function may limit the set of processing providers from whichit requests bids based on present server farm workloads, software orhardware limitations, green factors, location factors, weather/gridfactors or other factors to which the GWAE has knowledge. Within theGWAE mitigate function, processing providers may have the ability totarget competitiveness to certain classes of customers and may havedifferent response templates for different submitting customers, typesof work, etc.

Integration into a Carbon Trading System

In another embodiment, the GWAE is enabled to handle carbon credittrading. Carbon credit trading is a rapidly evolving system wherecompanies can purchase carbon credits to offset activities that producecarbon pollution above a threshold and/or earn carbon credits whenactivities produce carbon pollution below a threshold. The net effect ofthe carbon credit system is to cap/manage the amount of carbon beingreleased to the environment and increase greener operation. For manyindustries that are data management or data processing intensive,primary and secondary energy used in processing IT workloads is asignificant percentage of their carbon output and represents anopportunity for carbon trading.

In a GWAE adapted for carbon credit trading (CCT), new workload requestsmay be audited for information on whether the submitting entity isparticipating in CCT. If the entity is not participating in CCT, orparticipation is not mandated, submission through the GWAE operates asdetailed above. If CCT is enabled, carbon credit (CC) information aboutthe submitter may be audited to obtain the present CC balance, with theGWAE acting as a bookkeeper for credits. Should a positive CC balanceexist, submission through the GWAE operates as detailed above, and theGWAE submits the workload based on cost, time and environmentalparameters associated with the workload request, and the carbon tradingcredit for the newly submitted workload may be either positive ornegative. The CC banking query may be based on only previous history, orinclude an estimate of carbon credit requirements should the requestedworkload be performed on a non-environmentally friendly server farm.Should a negative balance exist, the GWAE will query the submitter, orsubmitter database, to determine if carbon credits should be purchasedto null the deficit, or estimated deficit. If CCs are purchased, the jobassignment continues as if a positive CC balance existed. If additionalCC purchases are refused, the GWAE limits submission opportunities forthe requested workload to only those servers that will produce a netpositive CC balance. Once a workload is submitted, the GWAE monitors thestatus of the workload and when the workload is completed, a calculationof CC earning/cost is made based on workload placement, run time,transmission costs etc., and GWAE book-keeping is updated accordingly.The GWAE CCT function operates to maintain a CCT neutral oralternatively, a CCT positive balance over time.

Referring to FIG. 7, there is shown a flow chart of the function 702 forhandling carbon credit trading. A request for carbon credit trading isreceived, (step 704), where it is determined if the request is a newrequest. If the request is not a new request, (NO; step 704), thefunction determines if the job is finished (step 706). If the job is notfinished (NO; step 706), the function returns to step 704. If the job isfinished (YES; step 706), the function calculates credit (step 708),adds the calculated credit to the balance (step 710), and returns tostep 704. Returning to step 704, if the request is a new request (YES;step 704), the function determines if the balance is positive (step712). When the balance is positive (YES; step 712) the function thenadvances to “submit to best matched server” (step 714) and then proceedsto step 704. Selection of the best matched server in step 714 may berestricted to nodes which result in addition to the positive creditbalance or may add or subtract from the balance dependent on submissionparameters for the workload. If the balance is not positive (NO; step712) the function advances to purchase credit (step 716). If credit isnot to be purchased (NO; step 716), the function advances to “submitonly to green server” (step 718) and then continues to step 704. If,however, credit is purchased (YES; step 716), the function purchasescredit (step 720), adds the credit to balance (step 722) and thenadvances to step 714.

In one embodiment of the invention jobs are assigned one at a time to aServer Farm (SF). In this embodiment, a job is taken off the queue andassigned to whichever server farm that best meets the criteria of thejob (e.g. jobs with green preferences are assigned to green servers).

Still another embodiment considers all jobs in the queue that have notbeen assigned to a server farm. An embodiment of this method uses linearprogramming to assign the jobs in combination with alternative jobassignment methods. The model formulation handles a single job on asingle processor (or server machine) at any given time and allows formultiple jobs within a time period as long as the total time spent bythe jobs in that time period does not exceed the duration of the period.This can be extended to machines with multiple processors (ability for amachine to handle multiple jobs) by treating the processors within themachine as individual machines which compete for the same resources.With this embodiment a set of jobs can be handled only by a specific setof servers (e.g. some jobs can be handled only by green servers). Thiscan be achieved either by the resources requirements of the jobs or jobpreferences.

The linear programming (LP) embodiment can be composed of an objectivefunction that defines a measure of the quality of a given solution, anda set of linear constraints. The types of equations used in jobassignment models can include:

-   -   1. Backorder Constraints which ensures that each job not        assigned in one period is backordered to the next period. (must        be assigned in a subsequent period)    -   2. Resource Capacity Constraints, which ensure that the capacity        available for processing job activities including capacity used        for jobs already assigned is not exceeded.    -   3. Assignment Constraints, each job is assigned to one server or        server family. The total time jobs spend in a period cannot        exceed the duration of the period.

An LP formulation which can be used is provided below in the formfamiliar to those practiced in the art; i.e., definition of subscripts,definition of objective function coefficients, definition of constants,definition of decision variables, LP formulation or equations.

Definition of Subscripts

j—Time periodm—Server or Server farm (could be a single processor)k—Job k (job arrival number)w—Server resource e.g. such as memory, disk space, etc.

Definition of Objective Function Coefficients

PRC_(mk)—cost of processing a job k on a server m.BC_(kj)—penalty for inability to assign job k by the end of period j.

Definition of Constants

-   REQ_(kw)—Resource requirements of job k and resource type w.-   RT_(km)—Run time of job k on server m.-   R_(jmw)—Total Resource for type w available for processing jobs that    have not yet been assigned to server m during time period j.-   BS_(j)—bucket size (BS) duration in period j (e.g. number hours in    period j).

Definition of LP Decision Variables

X_(jkm)—Assign job k to SF m in period j. (binary, takes values 0 or 1)B_(jk)—Backorder of job k in period j. B_(jk)=0 if job k assigned byperiod j; 1 otherwiseY_(jkm)—Total time job k spends running on server m in period j.

LP Equations or Formulation

The following minimizes the objective function subject to theconstraints shown below.

Objective Function: Minimize:

${\sum\limits_{j}{\sum\limits_{m}{\sum\limits_{k}{{PRC}_{mk}X_{jkm}}}}} + {\sum\limits_{j}{\sum\limits_{k}{B_{jk}{BC}_{jk}}}}$

Subject to: Backorder Constraints:

Ensures that B_(jk)=0 if job k assigned by period j, 1 otherwise

${B_{jk} = {B_{{({j - 1})}k} - {\sum\limits_{m}{X_{jkm}{\forall j}}}}},k$

j=Time period (j=1, 2 . . . N where N is number of time periods) andwhere

B_(0k)=1

Capacity Constraints:

REQ _(kw) *X _(jkm) ≦R _(mjw) ∀j,w,k,m

Assignment Constraint: Each job is assigned once.

${\sum\limits_{j}{\sum\limits_{m \in {S{(k)}}}X_{jkm}}} = {1{\forall k}}$

Where S(k) is the set of servers that can handle job k

The following two constraints ensure that once a job has begunprocessing on a server it completes in the earliest possible period andthat the total processing time at a server does not exceed the timeavailable:

${{\sum\limits_{r = v}^{v + {\lceil\frac{{RT}_{k}}{BS}\rceil}}Y_{rkm}} = {X_{vkm}{RT}_{k}{\forall k}}},m$

Where BS is average period size between periods v and

$v + \left\lceil \frac{{RT}_{k}}{BS} \right\rceil$

${{\sum\limits_{k}Y_{jkm}} \leq {{BS}_{j}{\forall j}}},m$

Non-Negativity Constraints:

all X_(i,j . . .) ≧0, where X is a generic decision variable and i, jetc. represent generic subscripts.

The model formulation above can be solved using an LP solver (such asCOIN or CPLEX solver) or a rule based heuristic.

The various method embodiments of the invention will be generallyimplemented by a computer executing a sequence of program instructionsfor carrying out the steps of the method, assuming all required data forprocessing is accessible to the computer. The sequence of programinstructions may be embodied in a computer program product comprisingmedia storing the program instructions. As will be readily apparent tothose skilled in the art, the present invention can be realized inhardware, software, or a combination of hardware and software. Any kindof computer/server system(s)—or other apparatus adapted for carrying outthe methods described herein—is suited. A typical combination ofhardware and software could be a general-purpose computer system with acomputer program that, when loaded and executed, carries out the method,and variations on the method as described herein. Alternatively, aspecific use computer, containing specialized hardware for carrying outone or more of the functional tasks of the invention, could be utilized.

As will be appreciated by one skilled in the art, the present inventionmay be embodied as a system, method or computer program product.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer-usableprogram code embodied in the medium.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM) or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then complied, interpreted, ofotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave, The computer usable program code may betransmitted using any appropriate medium, including but not limited towireless, wired, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, though the Internet using an Internet Service Provider).

The present invention is described above with reference to flow chartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flow chart illustrationsand/or block diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions meanswhich implement the function/act specified in the flowchart and/or blockdiagram block of blocks.

The computer program instruction may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Although a few examples of the present invention have been shown anddescribed, it would be appreciated by those skilled in the art thatchanges might be made in these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

1. A method of allocating a job submission for a computational task to aset of distributed server farms each having at least one processingentity comprising; receiving a workload request from at least oneprocessing entity for submission to at least one of the set ofdistributed server farms; using at least one or more conditionsassociated with the computational task for accepting or rejecting atleast one of the server farms to which the job submission is to beallocated; determining a server farm that can optimize said one or moreconditions; and dispatching the job submission to the server farm whichoptimizes the at least one of the one or more conditions associated withthe computational task and used for selecting the at least one of theserver farms.
 2. The method of claim 1 further comprising: using as acondition, auditing the workload request for environmental, monetary ortiming requirements; and calculating the environmental, monetary and/ortime costs for each server farm.
 3. The method of claim 1 furthercomprising: using as a condition, at least one of the lowestenvironmental impact as measured by a power source used to power theserver farm, a server farm power performance, and/or an expectedprocessing time of the workload and energy requirements for transportingthe workload to the server farm for rejecting a server farm.
 4. Themethod of claim 1 further comprising: using as a condition costsassociated with the computational task in addition to capability,availability and/or efficiency of a server farm.
 5. The method of claim1 wherein a server farm implements a carbon trading system whichcalculates carbon credits as a function of CO and/or CO₂ leveldifferentials between the energy source used and a base CO2 level usedto calculate a carbon credit/debit, said method using as a condition anamount of carbon credits for selecting a server farm.
 6. The method ofclaim 1 further comprising: using as a condition, power suppliersoperating either from renewable resources or non-renewable sources. 7.The method of claim 6 further comprising: using as a condition, thegeneration efficiency and environmental footprint per unit of energyproduced.
 8. The method of claim 1 further comprising: estimating whenpeak energy requirements of the geographical area where a server farm islocated is close to the maximum generation/transmission capacity of thegeographical area, and said method using as a condition said estimate.9. A system of allocating a job submission for a computational task to aset of distributed server fauns each having at least one processingentity comprising; receiving means for receiving a workload request fromat least one processing entity for submission to at least one of the setof distributed server farms; selecting means using at least one or moreconditions associated with the computational task for accepting orrejecting at least one of the server farms to which the job submissionis to be allocated; determining means for providing a server farm thatcan optimize said one or more condition; and sending means fordispatching the job submission to the server farm which optimizes the atleast one of the one or more conditions associated with thecomputational task and used for selecting the at least one of the serverfarms.
 10. The system of claim 9 further comprising: using as acondition, auditing workload request for environmental, monetary ortiming requirements; and calculating the environmental, monetary and/ortime costs for each server farm.
 11. The system of claim 9 furthercomprising: using as a condition as least one of the lowestenvironmental impact as measured by a power source used to power theserver farm, a server farm power performance, and/or an expectedprocessing time of the workload and energy requirements for transportingthe workload to the server farm for rejecting a server farm.
 12. Thesystem of claim 9 further comprising: using as a condition costsassociated with the computational task in addition to capability,availability and/or efficiency of a server farm.
 13. The system of claim9 wherein a server farm implements a carbon trading system whichcalculates carbon credits as a function of CO and/or CO2 leveldifferentials between the energy source used and a base CO2 level usedto calculate a carbon credit/debit, said method using as a condition anamount of carbon credits for selecting a server farm.
 14. The system ofclaim 9 further comprising: using as a condition, power suppliersoperating either from renewable resources or non-renewable sources. 15.The system of claim 14 further comprising: using as a condition, thegeneration efficiency and environmental footprint per unit of energyproduced.
 16. The system of claim 9 further comprising: estimating whenpeak energy requirements of the geographical area where a server farm islocated is close to the maximum generation/transmission capacity of thegeographical area, and said method using as a condition said estimate.17. A computer program product for use with a computer, the computerprogram product including a computer readable medium having recordedthereon a computer program or program code for causing the computer toperform a method for storing and retrieving data, the method comprising:receiving a workload request from at least one processing entity forsubmission to at least one of the set of distributed server farms; usingat least one or more conditions associated with the computational taskfor accepting or rejecting at least one of the server farms to which thejob submission is to be allocated; determining a server farm that canoptimize said one or more condition; and dispatching the job submissionto the server farm which optimizes the at least one of the one or moreconditions associated with the computational task and used for selectingthe at least one of the server farms.
 18. The computer program productof claim 17 further comprising: using as a condition, auditing workloadrequest for environmental, monetary or timing requirements; andcalculating the environmental, monetary and or time costs for eachserver farm.
 19. The computer program product of claim 18 furthercomprising: using as a condition, the lowest environmental impact asmeasured by a power source used to power the server farm, a server farmpower performance, and/or an expected processing time of the workloadand energy requirements for transporting the workload to the server farmfor rejecting a server farm.
 20. The computer program product of claim18 wherein a server farm implements a carbon trading system whichcalculates carbon credits as a function of CO and/or CO2 leveldifferentials between the energy source used and a base CO2 level usedto calculate a carbon credit/debit, said method using as a condition anamount of carbon credits for selecting a server farm.
 21. The computerprogram product of claim 17 further comprising: using as a condition,power suppliers operating either from renewable resources ornon-renewable sources.
 22. The computer program product of claim 21further comprising: using as a condition, the generation efficiency andenvironmental footprint per unit of energy produced.
 23. The computerprogram product of claim 17 further comprising: estimating when peakenergy requirements of the geographical area where a server farm islocated is close to the maximum generation/transmission capacity of thegeographical area, and said method using as a condition said estimate.